Optical receipt processing

ABSTRACT

Techniques for providing improved optical character recognition (OCR) for receipts are discussed herein. Some embodiments may provide for a system including one or more servers configured to perform receipt image cleanup, logo identification, and text extraction. The image cleanup may include transforming image data of the receipt by using image parameters values that optimize the logo identification, and performing logo identification using a comparison of the image data with training logos associated with merchants. When a merchant is identified, a second image clean up may be performed by using image parameter values optimized for text extraction. A receipt structure may be used to categorize the extracted text. Improved OCR accuracy is also achieved by applying on format rules of the receipt structure to the extracted text.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 17/115,447, filed on Dec. 8, 2020, which claimspriority to U.S. patent application Ser. No. 16/254,040, filed on Jan.22, 2019, which claims priority to and the benefit of U.S. patentapplication Ser. No. 15/281,517, filed Sep. 30, 2016, now U.S. Pat. No.10,229,314, which claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/235,173, filed Sep. 30, 2015, each titled“Optical Receipt Processing,” and the contents of each of which arehereby incorporated by reference in their entirety.

FIELD

Embodiments of the invention relate, generally, to techniques forproviding messages based on programmatic optical recognition.

BACKGROUND

Optical character recognition (OCR) refers to the conversion of an imageincluding text into machine-encoded text. For example, financial systemsuse OCR to extract check data from paper checks based on scanning thepaper check to generate image data, such as with a camera, and thenprocessing the image to extract text data of the check (e.g., amount,account number, etc.). As such, checks are typically created tofacilitate the OCR process. For example, checks use an OCR-A font thatis monospaced and simple thick strokes to support machine recognition.Checks also include standardized locations for data fields. The payeeand numerical dollar amount are typically on a first line, then thewritten amount on a subsequent line, then a signature block on a thirdline, and account numbers at the bottom. Using these standardizedlocations of check data on the check, each “field” of the check isprogrammatically extracted and categorized such that the check can beunderstood and processed. However, not all images are designed to besusceptible to OCR. Receipts of various merchants, for example, may usedifferent fonts and formats such that the conventional techniques usedin check processing fail to yield reliable data. In this regard andothers as discussed herein, areas for improving current techniques havebeen identified.

BRIEF SUMMARY

Through applied effort, ingenuity, and innovation, solutions to improvesuch systems have been realized and are described herein.

Various embodiments of the present invention are directed to improvedapparatuses, methods, and computer readable media for

Some embodiments may include methods, while other embodiments mayinclude circuitry and/or media configured to implement the methodsand/or other functionality discussed herein. For example, one or moreprocessors, and/or other machine components may be configured toimplement the functionality discussed herein based on instructionsand/or other data stored in memory and/or other non-transitory computerreadable media.

These characteristics as well as additional features, functions, anddetails of various embodiments are described below. Similarly,corresponding and additional embodiments are also described below.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described some embodiments in general terms, reference willnow be made to the accompanying drawings, which are not necessarilydrawn to scale, and wherein:

FIG. 1 shows an example of a system in accordance with some embodiments;

FIG. 2 shows an example of a system in accordance with some embodiments;

FIG. 3 shows a schematic block diagram of example circuitry inaccordance with some embodiments;

FIG. 4 shows a flow chart of an example of a method for receipt imagecleanup in accordance with some embodiments.

FIG. 5 shows an example of a method for receipt logo detection inaccordance with some embodiments;

FIG. 6 shows an example of a method for receipt text extraction inaccordance with some embodiments;

FIG. 7 shows an example of image data defining a receipt in accordancewith some embodiments;

FIG. 8 shows an example of image data programmatically transformed forreceipt detection and/or logo identification in accordance with someembodiments;

FIG. 9 shows an example of image data programmatically transformed fortext extraction in accordance with some embodiments;

FIG. 10 shows an example of a receipt structure in accordance with someembodiments; and

FIG. 11 shows an example of receipt text data extracted from image datain accordance with some embodiments.

DETAILED DESCRIPTION

Some embodiments of the present invention will now be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the invention are shown. Indeed, theinvention may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. Like numbers refer to like elements throughout.

As used herein, the terms “data,” “content,” “information,” and similarterms may be used interchangeably to refer to data capable of beingtransmitted, received, and/or stored in accordance with embodiments ofthe present invention. Thus, use of any such terms should not be takento limit the spirit and scope of embodiments of the present invention.Further, where a computing device is described herein to receive datafrom another computing device, it will be appreciated that the data maybe received directly from the another computing device or may bereceived indirectly via one or more intermediary computing devices, suchas, for example, one or more servers, relays, routers, network accesspoints, base stations, hosts, and/or the like, sometimes referred toherein as a “network.” Similarly, where a computing device is describedherein to send data to another computing device, it will be appreciatedthat the data may be sent directly to the another computing device ormay be sent indirectly via one or more intermediary computing devices,such as, for example, one or more servers, relays, routers, networkaccess points, base stations, hosts, and/or the like.

As used herein, the terms “provider” and “merchant” may be usedinterchangeably and may include, but are not limited to, a businessowner, consigner, shopkeeper, tradesperson, vendor, operator,entrepreneur, agent, dealer, organization or the like that is in thebusiness of a providing a good, service or experience to a consumer,facilitating the provision of a good, service or experience to aconsumer and/or otherwise operating in the stream of commerce. The“provider” or “merchant” need not actually market a product or servicevia the promotion and marketing service, as some merchants or providersmay utilize the promotion and marketing service only for the purpose ofgathering marketing information, demographic information, or the like.

As used herein, the term “consumer” should be understood to refer to arecipient of goods, services, promotions, media, or the like provided bythe promotion and marketing service and/or a merchant. Consumers mayinclude, without limitation, individuals, groups of individuals,corporations, other merchants, and the like.

As used herein, the term “image data” or “receipt image data” refers todata defining an image of a receipt. The receipt may be generated in thecourse of a transaction with a merchant, such as by a point-of-sale(POS) device of the merchant. The image data may be generated by acamera or other optical image capture device, such as a camera of amobile consumer device. The receipt may include “fields” or locationswhere text having predefined meanings (e.g., merchant name, transactiontime, shop location, return deadline, and item data, etc.) may beextracted.

As used herein, the term “receipt text data” refers to machine datadefining text that has been extracted from the image data using OCR. Thereceipt text data may include alphanumeric characters.

As used herein, the term “structured receipt data” refers to receipttext data that has been defined with respect to a receipt structure. Areceipt structure may include receipt elements corresponding with“fields” of data on the receipt. For example, a receipt element may bedefined for merchant name, transaction time, shop location, returndeadline, and item data (e.g., with receipt elements including item code(e.g., stock keeping unit (SKU) number), item name, item cost, etc.). Insome embodiments, different merchants may be associated with differentreceipt structures, each including receipt elements that are associatedwith locations where the corresponding fields are printed on thereceipt.

System Architecture and Example Apparatus

Methods, apparatuses, and computer program products of the presentinvention may be embodied by any of a variety of devices. For example,the method, apparatus, and computer program product of an exampleembodiment may be embodied by a networked device, such as a server orother network entity, configured to communicate with one or moredevices, such as one or more client devices. Additionally oralternatively, the computing device may include fixed computing devices,such as a personal computer or a computer workstation. Still further,example embodiments may be embodied by any of a variety of mobileterminals, such as a portable digital assistant (PDA), mobile telephone,smartphone, laptop computer, tablet computer, or any combination of theaforementioned devices.

In this regard, FIG. 1 shows an example computing system within whichembodiments of the present invention may operate. Consumers andmerchants may access a receipt processing service from a receiptprocessing system 102 via a network 112 (e.g., the Internet, or thelike) using computer devices 108A through 108N and 110A through 110N,respectively (e.g., one or more consumer devices 108A-108N or one ormore merchant devices 110A-110N). Moreover, the promotion and marketingsystem 102 may comprise a server 104 and a database 106.

The server 104 may be embodied as a single computer or multiplecomputers. The server 104 may provide for receiving of receipt imagedata from various sources, including but not necessarily limited to theconsumer devices 108A-108N and the merchant devices 110A-110N. Forexample, the server 104 may be operable to receive receipt image datafrom a consumer device 108, and process receipt image data. As discussedin greater detail herein, the server 104 may be configured to performimage cleanup (e.g., if the cleanup has not already been performed bythe consumer deice 108), detect a logo of a merchant from the receipt toidentify a merchant, and extract receipt text data, and then generatestructured receipt data with the extracted receipt text data and areceipt structure associated with the merchant. The structured receiptdata may then be used to facilitate commercial activities with merchantsand consumers, such as by building consumer profiles/preferences fortargeted advertising and/or tracking merchant inventory levels toimprove merchant operations. Although a single server 104 is shown,system 102 may include one or more servers 104. In some embodiments, theone or more servers 104 may include receipt log detector 212 and receipttext extractor 214, as shown in FIG. 2 . In some embodiments, server 104may further include a receipt cleanup 210.

Returning to FIG. 1 , database 106 may be embodied as a data storagedevice such as a Network Attached Storage (NAS) device or devices, or asa separate database server or servers. The database 106 includesinformation accessed and stored by the server 104 to facilitate theoperations of the receipt processing system 102. For example, thedatabase 106 may include, without limitation, image data, logo imagedata, training logo image data, receipt text data, structured receiptdata, receipt structures, merchant item data, consumer profile data,user account credentials, analytic results, reports, transaction data,and/or the like.

The consumer devices 108A-108N may be any computing device as known inthe art and operated by a consumer. Electronic data received by theserver 104 from the consumer devices 108A-108N may be provided invarious forms and via various methods. The consumer devices 108A-108Nmay include mobile devices, such as laptop computers, smartphones,netbooks, tablet computers, wearable devices (e.g., electronic watches,wrist bands, glasses, etc.), and the like. Such mobile devices mayprovide their receipt image data to the server 104 and receive targetedelectronic marketing communications, rewards, incentives, etc. that arerelevant to the sent data. As such, consumer device 108 may include acamera. In some embodiments, the consumer devices 108A-108N may includewired or stationary devices such as desktop computers or workstations.

In embodiments where a consumer device 108 is a mobile device, such as asmart phone or tablet, the consumer device 108 may execute an “app” or“consumer application” to interact with the receipt processing system102, such as a receipt processing application. Such apps are typicallydesigned to execute on mobile devices, such as tablets or smartphones.For example, an app may be provided that executes on mobile deviceoperating systems such as Apple Inc.'s iOS®, Google Inc.'s Android®, orMicrosoft Inc.'s Windows 8®. These platforms typically provideframeworks that allow apps to communicate with one another and withparticular hardware and software components of mobile devices. Forexample, the mobile operating systems named above each provideframeworks for interacting with location services circuitry, wired andwireless network interfaces, user contacts, and other applications in amanner that allows for improved interactions between apps while alsopreserving the privacy and security of consumers. In some embodiments, amobile operating system may also provide for improved communicationinterfaces for interacting with external devices (e.g., home automationsystems, indoor navigation systems, and the like). Communication withhardware and software modules executing outside of the app is typicallyprovided via application programming interfaces (APIs) provided by themobile device operating system.

Various other types of information may also be provided in conjunctionwith an app executing on the consumer's mobile device. For example, ifthe mobile device includes a social networking feature, the consumer mayenable the app to provide updates to the consumer's social network tonotify friends of a particularly interesting promotion. It should beappreciated that the use of mobile technology and associated appframeworks may provide for particularly unique and beneficial uses ofthe promotion and marketing service through leveraging the functionalityoffered by the various mobile operating systems.

Additionally or alternatively, the consumer device 108 may interactthrough the receipt processing system 102 via a web browser. As yetanother example, the consumer device 108 may include various hardware orfirmware designed to interface with the receipt system 102 (e.g., wherethe consumer device 108 is a purpose-built device offered for theprimary purpose of communicating with the receipt processing system 102,such as a store kiosk).

The merchant devices 110A-110N may be any computing device as known inthe art and operated by a merchant. For example, the merchant devices110A-110N may include a merchant point-of-sale configured to facilitatetransactions at the merchant shop and generate receipts includingtransaction data associated with the transactions.

FIG. 2 shows an example of a system 100 in accordance with someembodiments. In particular, FIG. 2 shows schematic block diagrams offunctional modules of the system 100. Consumer device 108 may include acamera 202 and a receipt cleanup 210. The consumer device 108 and system102 may be connected with each other via the network 112.

System 102 may include receipt logo detector 212 and receipt textextractor 214, which may be implemented via the one or more servers 104and one or more databases 106 in some embodiments. In some embodiments,the server 102 may include the receipt cleanup module 210.

The receipt cleanup 210 may be configured to receive receipt image dataof a receipt 204 captured by camera 202, and extract receipt text datafrom the image data. Subsequent to a cleanup, the receipt cleanup 210may be configured to provide the image data to the receipt logo detector212. The receipt logo detector 212 may be configured to detect a logofrom the receipt image data, and then identify a merchant from the logobased on a comparison with training logo data. Subsequent to merchantidentification, the receipt text extractor 214 may be configured toextract receipt text data from the image data, and generate structuredreceipt data on associating the receipt text data with receipt elementsof a receipt structure associated with the merchant. In someembodiments, the receipt text extractor 214 may be further configured todetermine format rules associated with the receipt elements, and update(e.g., correct) the structured receipt data based on the format rules.In some embodiments, a merchant device 110 may also include a cameraand/or receipt cleanup 210. In that sense, the discussion herein withrespect to providing receipt processing for consumer devices 108 mayalso be applicable for merchant devices 110.

Example Apparatus[es] for Implementing Various Embodiments

The server 104, database 106, consumer device 108 or merchant device 110may be embodied by one or more computing systems or devices, such asapparatus 300 shown in FIG. 3 . As illustrated in FIG. 3 , the apparatus300 may include a processor 302, a memory 304, an input/output circuitry306, communications circuitry 308, a receipt cleanup circuitry 210, areceipt logo detector circuitry 212, and/or a receipt text extractor214. The apparatus 300 may be configured to execute the operationsdescribed herein. Although these components are described with respectto functional limitations, it should be understood that the particularimplementations necessarily include the use of particular hardware. Itshould also be understood that certain of these components may includesimilar or common hardware. For example, two sets of circuitry may bothleverage use of the same processor, network interface, storage medium,or the like to perform their associated functions, such that duplicatehardware is not required for each set of circuitry. The use of the term“circuitry” as used herein with respect to components of the apparatusshould therefore be understood to include particular hardware configuredto perform the functions associated with the particular circuitry asdescribed herein.

The term “circuitry” should be understood broadly to include hardwareand, in some embodiments, software for configuring the hardware. Forexample, in some embodiments, “circuitry” may include processingcircuitry, storage media, network interfaces, input/output devices, andthe like. In some embodiments, other elements of the apparatus 300 mayprovide or supplement the functionality of particular circuitry. Forexample, the processor 302 may provide processing functionality, thememory 304 may provide storage functionality, the communicationscircuitry 308 may provide network interface functionality, and the like.

In some embodiments, the processor 302 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 304 via a bus forpassing information among components of the apparatus 300. The memory304 may be non-transitory and may include, for example, one or morevolatile and/or non-volatile memories. In other words, for example, thememory may be an electronic storage device (e.g., a computer readablestorage medium). The memory 304 may be configured to store information,data, content, applications, instructions, or the like, for enabling theapparatus to carry out various functions in accordance with exampleembodiments of the present invention.

The processor 302 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. Additionally or alternatively, the processor mayinclude one or more processors configured in tandem via a bus to enableindependent execution of instructions, pipelining, and/ormultithreading. The use of the term “processing circuitry” may beunderstood to include a single core processor, a multi-core processor,multiple processors internal to the apparatus, and/or remote or “cloud”processors.

In an example embodiment, the processor 302 may be configured to executeinstructions stored in the memory 304 or otherwise accessible to theprocessor. Alternatively or additionally, the processor may beconfigured to execute hard-coded functionality. As such, whetherconfigured by hardware or software methods, or by a combination thereof,the processor may represent an entity (e.g., physically embodied incircuitry) capable of performing operations according to an embodimentof the present invention while configured accordingly. Alternatively, asanother example, when the processor is embodied as an executor ofsoftware instructions, the instructions may specifically configure theprocessor to perform the algorithms and/or operations described hereinwhen the instructions are executed.

In some embodiments, the apparatus 300 may include input/outputcircuitry 306 that may, in turn, be in communication with processor 302to provide output to the user and, in some embodiments, to receive anindication of a user input. The input/output circuitry 306 may comprisea user interface and may include a display and may comprise a web userinterface, a mobile application, a client device, a kiosk, or the like.Where the input/output circuitry 306 is part of a consumer device 108,the input/output circuitry may include a camera or other optical imagecapturing device. In some embodiments, the input/output circuitry 306may also include a keyboard, a mouse, a joystick, soft keys, amicrophone, motion sensor, touch screen, a speaker, or otherinput/output mechanisms. The processor and/or user interface circuitrycomprising the processor may be configured to control one or morefunctions of one or more user interface elements through computerprogram instructions (e.g., software and/or firmware) stored on a memoryaccessible to the processor (e.g., memory 304, and/or the like).

The communications circuitry 308 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 300. In this regard, the communications circuitry 308may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 308 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally or alternatively,the communication interface may include the circuitry for interactingwith the antenna(s) to cause transmission of signals via the antenna(s)or to handle receipt of signals received via the antenna(s).

When circuitry 300 is implemented on a server 104, circuitry 300 mayinclude the receipt logo detector circuitry 212 and receipt textextractor circuitry 214. When circuitry 300 is implemented on a consumerdevice 108, circuitry 300 may include the receipt cleanup circuitry 214.

Circuitries 210, 212, and 214 may utilize processing circuitry, such asthe processor 302, to perform these actions. However, it should also beappreciated that, in some embodiments, circuitries 210, 212, and 214 mayinclude a separate processor, specially configured field programmablegate array (FPGA), or application specific interface circuit (ASIC).Circuitries 210, 212, and 214 may therefore be implemented usinghardware components of the apparatus configured by either hardware orsoftware for implementing these planned functions.

As will be appreciated, any such computer program instructions and/orother type of code may be loaded onto a computer, processor or otherprogrammable apparatus's circuitry to produce a machine, such that thecomputer, processor other programmable circuitry that execute the codeon the machine create the means for implementing various functions,including those described herein.

It is also noted that all or some of the information presented by theexample displays discussed herein can be based on data that is received,generated and/or maintained by one or more components of apparatus 300.In some embodiments, one or more external systems (such as a remotecloud computing and/or data storage system) may also be leveraged toprovide at least some of the functionality discussed herein.

As described above and as will be appreciated based on this disclosure,embodiments of the present invention may be configured as methods,mobile devices, backend network devices, and the like. Accordingly,embodiments may comprise various means including entirely of hardware orany combination of software and hardware. Furthermore, embodiments maytake the form of a computer program product on at least onenon-transitory computer-readable storage medium having computer-readableprogram instructions (e.g., computer software) embodied in the storagemedium. Any suitable computer-readable storage medium may be utilizedincluding non-transitory hard disks, CD-ROMs, flash memory, opticalstorage devices, or magnetic storage devices.

Receipt Processing

FIG. 4 shows a flow chart of an example of a method for receipt imagecleanup in accordance with some embodiments. Method 400 may begin at 402and proceed to 404, where one or more servers 104 (e.g., of the receiptprocessing system 102) may be configured to send a request for imagedata defining an image of a receipt to a consumer device 108.

The one or more may establish a network connection with a consumerdevice for transfer of the image data. For example, the connection maybe established based on the consumer device 108 connecting with the oneor more servers 104 of the receipt processing system 102 via network112, as shown in FIG. 1 .

In some embodiments, the network connection may be initiated based onconsumer device access to a consumer application for providing aconsumer interface installed and/or otherwise executing on the consumerdevice. In another example, the consumer device may access a webpageconfigured to provide the consumer application/interface, such as byentering a uniform resource locator (URL) within a web browser. Theconnection between the consumer device 108 and the one or more servers104 may include a wireless connection such that the consumer device maymaintain network connectivity while traveling to different locations.The consumer application may be installed on and/or otherwise executedby the processing circuitry of the consumer device, such as on a mobiledevice operating system as discussed above. Subsequent to the consumeraccessing the consumer application or webpage, the server 104 may beconfigured to send the request for the image data.

At 406, the consumer device 108 may be configured to create the imagedata with a camera. FIG. 7 shows an example of image data 700 defining areceipt 702, in accordance with some embodiments. Receipt 702 has beencaptured in front of a background 704, and includes a receipt logo 706and text 708. The image data 700 may be captured by the camera, whichmay be an integrated device of the consumer device 108, or may beotherwise provided to the consumer device 108, such as from a separatecamera or other image capturing device. The image may be created basedon the consumer pointing the camera to the physical receipt, andcapturing an image of the receipt.

At 408, the consumer device 108 (e.g., the receipt cleanup 210) may beconfigured to perform image cleanup for logo identification. The logoidentification may also include a receipt identification where thereceipt is extracted from the background to facilitate the logoidentification. In some embodiments, the one or more servers 104 mayinclude the receipt cleanup 210. Here, the consumer device 108 may beconfigured to send the image data to the one or more servers 104subsequent to creation by the camera, and without performing an imagecleanup. However, performing the image cleanup on the consumer device108 is advantageous for the technical problem of reducing central serverprocessing load, such as when the server 104 is providing receiptprocessing concurrently to multiple consumer devices.

The image cleanup may include performing an algorithmic transformationof the pixels of the image based on setting one or more image parametersto predefined values. Example image parameters may include contrast,brightness, filter size, offset, smoothing strategy, enhancementstrategy, rotations/skew, etc. Image cleanup may be performed tofacilitate receipt identification from the background of the image,and/or to facilitate logo identification.

Filter size refers to the size of the image filter used to clean noisefrom the background of the image (in pixels). Offset refers to thespacing of the filter used to clean noise from the background. Smoothingthreshold refers to the threshold of color difference between pixels fortext smoothing. If pixels have colors within this difference threshold,they will be smoothed into the same color to make text more clear.Enhancement strategy enhances the contrast of the image to normalize thebrightest and darkest pixels. Rotation/skew refers to the orientation ofthe image. Some other example image parameter adjustments may includeincreasing sharpness, adaptive blur, or background color.

FIG. 8 shows an example of image data 800 programmatically transformedfor receipt detection and/or logo identification in accordance with someembodiments. The contrast of the image data 700 may be iterativelyadjusted until the edges 804 of the transformed receipt 402 areidentified within the image data 800. For a white receipt on a darkbackground, little iteration may be needed. However, in an example ofthe white receipt on a light background, more iterations of (e.g.,increasing) contrast may be needed to identify the edges of the receipt.The receipt cleanup 210 may be configured to perform a pixel-by-pixelanalysis of each adjusted image to identify a white and/or rectangularshape that is common to nearly all commercial receipts. However, othertypes of colors and shapes may be used as parameters for thepixel-by-pixel analysis as could be necessitated by merchant use.

In some embodiments, the receipt cleanup 210 may be configured toperform a predefined number (e.g., 30) iterations of contrast imageparameter values in the event that the receipt edges (and/or a logo) arenot identified, and send a request to the display of the consumer device108 to capture a second image of the receipt if the receipt edges arenot identified subsequent to the predefined number of iterations.

In some embodiments, the contrast of a digital image may represent theamount of differences between black and white in an image. The pixels ofthe image may define a data range and a display range. The data range isthe range of intensity values used in the image (e.g., from 0 to 255),while the display range is the black-to-white mapping used to displaythe image (e.g., also mapped from 0 to 255). The contrast may representthe display range, with larger contrast representing a larger displayrange and lower contrast representing a lower display range. Forexample, a contrast of 255 may represent the highest display range value(between 0 and 255), where the color black is mapped to the darkest datarange value in the image (e.g., 255 if the image includes black), andthe rest of the image intensities are scaled accordingly until white ismapped to 0. In another example, a contrast of 150 would result in theintensity values above 150 (e.g., from 150 to 255) being transformedinto pure black, and the rest of the image intensities scaledaccordingly. In some embodiments, the lower display range mayadditionally or alternatively be adjusted to increase or decrease thecontrast image parameter value.

When contrast values scale from 0 to 255, the iterative testing forreceipt or logo detection may be scaled accordingly. For example, thecontrast image parameter value may be set to 5, the image may betransformed, and receipt/logo detection may be performed. The contrastimage parameter value may then be incremented by 8, such as for eachsuccessive image transformation, to result in 30 test iterations ofcontrast image parameter values between 0 and 255. It is appreciatedthat the display range of 0 and 255 is only an example, and the scalingcould be adjusted (e.g., between 0 and 30, to provide a unitary stepsize per iteration).

The brightness image parameter value may also be iteratively tested foroptimization. The brightness of an image pixel may represent the averageof the RGB color values, or the lightness parameter of a cylindricalcoordinate representation of RGB color values (e.g., HSL, HSV, HSB,etc.). In some embodiments, the brightness image parameter value may bedefined from 0 to 1. Here, iterative tests may be performed, such as 30iterative tests with uniform step sizes between 0 and 1. In someembodiments, the testing may be optimized by using image parameter valueranges in between the possible extremes, such as between 150 and 255 forthe contrast image parameter value because the lowest contrasts may beexpected to obscure some visual data.

Advantageously, the process of iterations and subsequent image capturemay be performed on the consumer device 108, with only image data havingsuccessfully identified receipts and/or logos (e.g., based on colorand/or shape definition) being sent to the server 104. As such, serverprocessing load is reduced. Network congestion that may otherwise becaused by the transfer (and retransfer) of unusable images from theconsumer device 108 to the server 104 via the network 114 is alsoreduced.

Subsequent to the identification of the edges of the receipt, thereceipt cleanup 210 may be configured to perform deskew, resize, and/ordewarp of the image. For example, the receipt cleanup 210 may beconfigured to perform deskew of the receipt based on the identifiededges. As such, a consistent orientation may be maintained for receiptimages to facilitate pixel-by-pixel comparisons/analysis as discussed ingreater detail herein. In some embodiments, deskew may be performedbased on logo identification, such as by deskewing the image based onthe deskew image parameter value that optimizes a logo match score.

Additionally or alternatively, the receipt cleanup 210 may be configuredto perform a resize of the receipt based on the identified edges. Forexample, merchant receipts typically are printed on standard orconsistently sized paper rolls, however, the size of the receipt on animage can vary such as based on camera distance when the image wastaken. As such, a consistent size for the receipt may be maintained tofacilitate the pixel-by-pixel comparisons/analysis.

Additionally or alternatively, the receipt cleanup 210 may be configuredto perform dewarp of the receipt based on the identified edges. Dewarprefers to the virtual straightening of curvature, folds, and/orcrumpling of the receipt paper to produce an image that likely moreaccurately represents the contents printed on the receipt than thedistorted image of the receipt itself.

At 410, the one or more server 104 (e.g., the receipt logo detector 212)may be configured to attempt logo identification from the (e.g., cleanedup) image data based on training logo image data. As discussed ingreater detail below with respect to method 500 and FIG. 5 , the logoidentification may be based on determining “logo match scores” betweenthe logo on the receipt and training logos defined by the training logoimage data. For example, the logo match score may indicate a level ofmatching for a transformed image logo 806 and a training logo, such asbased on a pixel-by-pixel comparison.

In some embodiments, the image cleanup and logo detection may include aniterative process wherein different image parameters are adjusted untila logo is successfully detected. For example, the receipt cleanup 210may be configured to utilize a genetic algorithm that iterates throughdifferent image parameter values, and determines logo match scores basedon a pixel-by-pixel comparison of the logo as transformed by thedifferent image parameter values and the training logos associated withmerchants. The iteration may include generating transformed images usingcombinations of different image parameter values (e.g., contrast,brightness, filter size, offset, smoothing strategy, enhancementstrategy, rotations/skew, etc.) that can be applied to transform theimage, then comparing each transformed image logo to each of thetraining logos.

The combinations of image parameter values may serve as the populationof candidate solution to the genetic algorithm to optimize the highestlogo match score across the training logos. An initial population may becreated based on random selections of image parameter valuecombinations, based on a prior best solution as determined by thegenetic algorithm (e.g., for a prior receipt or image captured by thecamera), predefined image parameter values, or the image parametervalues that optimized receipt detection as discussed above at 408. Themore “fit” initial combinations of image parameter values may beselected (e.g., stochastically), and modified (e.g., such as by randomchange of an image parameter value and/or a crossover of image parametervalues) to create the next generation, and so forth iteratively until anoptimal solution is achieved for the image parameter values. The geneticalgorithm may be performed for multiple test images (e.g., a seed of100) of receipts, and the most optimized image parameter values acrossthe test images may be determined, and used for subsequent image cleanup.

In some embodiments, the iteratively determined image parameter valuesresulting in the highest logo match scores may be used as predefinedimage parameter values, such as for a subsequent image cleanup for logoand/or receipt identification. Advantageously, the iterative imageparameter value optimization may be performed based on one or morereceipts captured by the camera of the consumer device 108, and thusprovides a technical solution to the technical problem of varying imagequality and characteristics (e.g., based on different camera designs,models, manufacturers, etc., or user variance such as picture takingdistance, angle, etc.).

As discussed in greater detail below, image cleanup can also beperformed for text extraction.

At 412, the one or more servers 104 (e.g., the receipt log detector 212)may be configured to determine whether a logo was identified. Forexample, the logo match scores may be compared with a predefined logomatch score threshold (e.g., 77%), and a logo may be identified asmatching a training logo of a merchant based on the associated logomatch score between the logo and the training logo exceeding the logomatch score threshold. In some embodiments, one or more image cleanupsmay be performed subsequent to the logo detection, such as size, deskew,or dewarp transformations.

In response to determining that the logo failed to be identified, method400 may return to 404, where the one or more servers 104 may be furtherconfigured to send a subsequent request for image data defining an imageof the receipt to the consumer device 108. As such, in some embodiments,steps 402-410 may be repeated (e.g., as deemed appropriate by theconsumer operating the consumer device 108) until a logo of a merchanthas been identified.

Returning to 412, in response to determining that merchant logo has beenidentified, method 400 may proceed to 414, where the one or more servers(e.g., the receipt log detector 212) may be configured to associate thereceipt with the merchant. In some embodiments, the association may bebased on the logo image and the name of the merchant, or an identifierof the merchant. As discussed in greater detail below with respect tomethod 600 and FIG. 6 , each merchant may be further associated with areceipt structure including receipt elements defined by the fields ofthe receipt of the merchant. Different merchants may be associated withdifferent receipt structures, each including receipt elements that areassociated with different physical locations of receipts of thedifferent merchants where the corresponding fields are printed.Advantageously, the use of the receipt structure may provide forimproved accuracy in text extraction by leveraging predefinedexpectations, or “format rules” associated with the receipt elements.

At 416, the one or more servers 104 (e.g., the receipt cleanup 210) maybe configured to perform image cleanup for text extraction. The imagecleanup techniques discussed at 408 may be applicable at 418. Forexample, the receipt cleanup 210 may be configured to perform analgorithmic transformation of the pixels of the image based on settingone or more image parameters to predefined values. The predefined imageparameter values may be different from the image parameter values usedin the image cleanup for logo/receipt identification. Alternatively, asingle set of predefined values may be used for both logo identificationand text extraction. Here, a single image cleanup step may be performed,such as at 408.

FIG. 9 shows an example of image data 900 programmatically transformedfor text extraction in accordance with some embodiments. The image data900 may include receipt 902 including logo 904 and text 906. The imagedata 900 represents a receipt image subsequent to transformation basedon image parameters optimized for text extraction. For example, tofacilitate point-by-point comparisons, the receipt image data has beenrotated, contrast and brightness increased, orientation of the receipthas been deskewed, and the background has been cropped out.

In some embodiments, determination of image parameter values for textextraction may include determining “text match scores” for combinationsof image parameter values. The accuracy of text extraction may bedetermined based on the text match scores, defining an amount of matchbetween extracted OCR text of a (e.g., training) receipt image data andknown keywords of the receipt. For example, the text match score may bedefined as a percent, with 100% being the highest score and meaning thatall corresponding characters of extracted text from the receipt imagedata and the characters of the keywords matched, and 0% being the lowestscore and meaning that no corresponding characters of the extracted textand keywords matched.

In some embodiments, the receipt cleanup 210 may be configured toutilize a genetic algorithm that uses a training receipt including knownkeywords. Here, image parameter values are optimized to improve accuracyof text extraction. The genetic algorithm may include iterating throughdifferent image parameter values, and determining text match scoresbased on a comparison of extracted text (e.g., receipt text data orstructured receipt data) with the known keywords of the receipt. Theiteration may include generating transformed images using combinationsof different image parameter values (e.g., contrast, brightness, etc.)that can be applied to transform the image, then the extracted receipttext with the known keywords of the receipt. In some embodiments, knownkeywords may be selected based on words that are expected to be found onreceipts such as “tax”, “TOTAL”, or “SUBTOTAL.” In some embodiments, apredefined number of image parameters (e.g., 5) may be used, such ascontrast, brightness, filter size, offset, smoothing strategy,enhancement strategy, rotations/skew, etc.

The combinations of image parameter values may serve as the populationof candidate solution to the genetic algorithm to optimize the highesttext match score. An initial population may be created based on randomselections of image parameter value combinations, based on a prior bestsolution as determined by the genetic algorithm (e.g., for a priorreceipt or image captured by the camera), predefined image parametervalues, or the image parameter values that optimized receipt detectionor logo identification. The more “fit” initial combinations of imageparameter values may be selected, and modified (e.g., such as by randomchange of an image parameter value and/or a crossover of image parametervalues) to create the next generation, and so forth iteratively until anoptimal solution is achieved for the image parameter values.

In some embodiments, the iteratively determined image parameter valuesresulting in the highest text match scores may be used as predefinedimage parameter values, such as for a subsequent image cleanup for textextraction. In some embodiments, the optimized image parameters for textextraction may be used as predefined values for the image parameters ina single image cleanup step (e.g., at 408). In some embodiments, asingle genetic algorithm-based image cleanup may be performed, such asto determine image parameters that optimize logo detection (e.g., basedon logo match scores) and text extraction (e.g., based on text matchscores). Accordingly, the receipt image data may be transformed based onthe optimized or predefined image parameters values.

At 418, the one or more servers 104 (e.g., the receipt text extractor214) may be configured to attempt text extraction based on a receiptstructure associated with the merchant. The text extraction may beperformed using OCR. As discussed in greater detail below in connectionwith method 600 and FIG. 6 , the identified merchant (e.g., based onlogo identification) may be associated with a receipt structureincluding receipt elements. Furthermore, the receipt text extractor 214may be configured to associate extracted receipt text data with thereceipt elements of the receipt structure to facilitate the textextraction.

At 420, the one or more servers 104 (e.g., the receipt text extractor214) may be configured to determine whether the text extraction issuccessful. A successful text extraction may result in the creation ofreceipt text data via OCR. In contrast, an unsuccessful text extractionmay result in a lack of usable receipt text data (e.g., no textextracted).

In some embodiments, the determination as to whether a text extractionwas successful may include a comparison of a text match score betweenthe receipt text data and known keywords associated with the receiptstructure, and a text match score threshold. The text match scorethreshold may define the text match score needed (e.g., 70%) between thereceipt text data and the known keywords of the receipt (e.g., “tax,”“TOTAL,” or “SUBTOTAL,” etc.). In some embodiments, the text match scorethreshold may be set to allow for different levels of tolerance fordifferent merchant receipts. For example, some receipts may use fontsthat are more susceptible to OCR character differentiation, while otherreceipts may use fonts with very similar, but different characters(e.g., the number 0 and the letter O). Thus each merchant or receiptstructure may also be associated with a text match score threshold.

In some embodiments, the determination of a text match score thresholdof a merchant's receipts may use a manual process to develop a trainingset. Through (e.g., online) crowd source labor or employees, the systemmay show receipt images alongside extracted structured receipt data, andrequest manual verification of key data points (e.g., compare withkeywords). The manual response indicates the accuracy of the OCR, andmay be used to set the text match score threshold. In another example,the text match score threshold may be determined programmatically, suchas to set a predetermined percentage (e.g., 80%) of text match scoresfor a sample of test images as being successfully extracted.

In response to determining that the text extraction fails to besuccessful, method 400 may return to 404, where the one or more servers104 may be further configured to send a subsequent request for imagedata defining an image of the receipt to the consumer device 108. Assuch, repeated attempts at image clean up and text extraction may beperformed for new receipt image data, such as until receipt text datahas been successfully extracted.

Returning to 420, in response to determining that merchant logo has beenidentified, method 400 may proceed to 422, where the one or more servers(e.g., the receipt text extractor 214) may be configured to generatestructured receipt data based on the receipt text data. The structuredreceipt data may include the extracted receipt text data categorizedbased on the field or subject matter of the text. As such, fine receiptdata including the details of individual items (e.g., name, SKU, price,quantity, etc.) may be programmatically extracted from the image data ofthe receipt. Additional details with respect to generating and updatingstructured receipt data are discussed below in connection with method600 and FIG. 6 . Method 400 may then proceed to 424 and end.

FIG. 5 shows an example of a method 500 for receipt logo detection inaccordance with some embodiments. Method 500 may begin at 502 andproceed to 504, where the one or more servers 104 (e.g., the receipt logdetector 212) may be configured to identify a log of a receipt based onimage data defining an image of the receipt.

The image data may be created with a camera of a consumer device 108, asdiscussed at 404, and may have been further processed using an imagecleanup, such as discussed at 408 and/or 416 of method 400. Through theimage cleanup, a transformed receipt image that is optimized for logoidentification may be created and used for the logo identification, suchas image data 800 shown in FIG. 8 . In some embodiments, method 500 maybe performed as part of an iterative and/or genetic algorithm forgenerating optimized image parameters using determinations of logo matchscores.

In some embodiments, subsequent to image cleanup, identification of thelogo or region where the logo is located within the receipt image datamay be performed based on applying a color and/or shape definition to asearch 5 of the pixels of the receipt image. For example, the image data800 may be analyzed to locate a cluster of (e.g., black) colored pixelsthat represents the logo 806. Subsequent to identifying the logo 806 onthe receipt 802, the pixels associated with the logo 806 may be used formerchant identification.

At 506, the one or more servers 104 (e.g., the receipt logo detector212) may be configured to determine training logo image data definingtraining logos associated with a merchant. The receipt processing system102 may include the database 106 configured to store training logoimages associated with merchants. The training logo data associationswith merchants may be predefined, for example, by a human operator thatcreates the training logo images from receipts of the merchants. Foreach unique merchant one or more training logos may be stored andaccessed to perform merchant detection based on the logo identification.In some embodiments, captured training logo image data may also beprocessed using image cleaning techniques prior to being stored and usedas logo identification training data examples.

At 508, the one or more servers 104 (e.g., the receipt logo detector212) may be configured to determine logo match scores between the logoof the receipt and a training logo associated with the merchant. A logomatch score may indicate a level of matching for a transformed (e.g.,via image clean up) image logo and the training logo of the merchant,such as based on a pixel-by-pixel comparison. In some embodiments, thelogo match score may be defined as a percent, with 100% being thehighest score and meaning that all corresponding pixels (e.g., based onX, Y pixel coordinates) of the transformed image logo and the traininglogo matched, and 0% being the lowest score and meaning that nocorresponding pixels of the two logos matched. Thus, the logo matchscore may be generated based on a pixel-by-pixel comparison of the logo,as transformed by the image parameter values, with the training logosassociated with merchants.

At 510, the one or more servers 104 (e.g., the receipt logo detector212) may be configured to determine a logo match score threshold. Thelogo match score threshold may define whether a logo match score betweenthe receipt image logo and the training logo indicates a match betweenthe receipt image logo and the training logo, and thus also a match forthe merchant associated with the training logo. A higher logo matchscore threshold nay be used to impose greater image data similarityrequirement for determining a successful match, while a lower logo matchscore threshold may be used to impose a lower image data similarityrequirement.

At 512, the one or more servers 104 (e.g., the receipt logo detector212) may be configured to determine whether the logo match scoresatisfies the logo match score threshold. For example, the logo matchscores may be compared with a predefined logo match score threshold(e.g., 77%), and a logo may be identified as matching the training logoof the merchant based on the associated logo match score between thelogo and the training logo exceeding the logo match score threshold.

In response to determining that the logo match score fails to satisfythe logo match score threshold, method 500 may return to 508, where theone or more servers 104 may be configured to determine a logo matchscore between the logo of the receipt and a second training logoassociated with the merchant. The second training logo may be one of aplurality of training logos associated with the merchant. In someembodiments, steps 508-512 may be repeated for each of the traininglogos associated with the merchant.

In some embodiments, when each of the training logos associated with themerchant has been tests (e.g., and no match was found) at 512, method500 may return to 506, where the one or more servers 104 may beconfigured to determine training logo image data defining one or moretraining logos associated with a second merchant. Until a logo match isfound, logo match scores may be determined for each training logo (e.g.,across all merchants). In some embodiments, the one or more servers 104may be configured to iterate through each of the training logos acrossall merchants to perform comparison of training logos.

In some embodiments, when multiple training logos exceed the logo matchscore threshold, the logo may be identified as matching the traininglogo associated with the highest logo match score. In another example,when no logo match score exceeds the logo match score threshold, the oneor more servers 104 may be configured to determine that the logo failedto be identified, or alternatively, select the highest logo match score.

Returning to 512, in response to determining that the logo match scoreof a training logo associated with a merchant satisfies the logo matchscore threshold, method 500 may proceed to 514, where the one or moreservers 104 (e.g., the receipt logo detector 212) may be configured todetermine the merchant associated the training logo as being associatedwith the receipt. As such, the identity of the merchant may bedetermined based on a programmatic visual analysis of the receipt.

At 516, the one or more servers 104 (e.g., the receipt logo detector212) may be configured to add the image of the logo to the training logoimage data associated with the merchant as a new training logo. Thus,the new training logo can be a dynamic addition to the initial pool ofrepresentative training logos of the merchant (e.g., as created by amanual operator).

At 518, the one or more servers 104 (e.g., the receipt logo detector212) may be configured to update the logo match score threshold based onnumber of training logos associated with the merchant. Advantageously, agreater number of training logos associated with a single merchantallows for improved accuracy and precision for logo identification basedon pixel-by-pixel comparisons. As such, the logo match score thresholdmay be larger when a merchant has a larger number of training logos, andsmaller when the merchant has a smaller number of training logos. When anew training logo is added, the logo match score threshold may beincreased. The updated logo match score threshold of the merchant maythen be used for subsequent logo identifications as discussed above at512.

Some embodiments may use different thresholds for logo identificationthan training logo addition. For example, a higher threshold may be setfor training logo addition than logo identification, such that onlycloser matching receipt images become part of the training logos of themerchant. In another example, a single threshold may be used such thatmore receipt images are added to the training logos, such as todynamically build a larger collection of representative examples thatcan improve the accuracy of logo identification. Method 500 may thenproceed to 520 and end.

FIG. 6 shows an example of a method 600 for receipt text extraction inaccordance with some embodiments. Method 600 may begin at 602 andproceed to 604, where one or more servers 104 (e.g., the receipt logodetector 212) may be configured to determine a merchant as beingassociated with a receipt based on logo detection. The discussions at414 and 514 of methods 400 and 500 discussed above may be applicable at602. Furthermore, the one or more servers 104 may be configured toperform and image cleanup, as discussed at 408 and 416 of method 400. Assuch, the receipt image may be in a form that is optimized for textextraction, such as image data 900 shown in FIG. 9 . Additionally oralternatively, determination of the merchant may be based on othertechniques, such as consumer device input by the consumer, or consumerdevice location detection (e.g., consumer is at a known merchantlocation).

At 606, the one or more servers 104 (e.g., the receipt text extractor214) may be configured to determine a receipt structure associated withthe merchant. The receipt structure may define receipt elementscorresponding with “fields” of data on the physical receipt. Eachreceipt element may also be associated with a location (e.g., X, Ycoordinate) that corresponds with the location of the correspondingfield on the physical receipt. For example, a receipt element may bedefined for merchant name, transaction time, shop location, returndeadline, and item data (e.g., with receipt elements including item code(e.g., stock keeping unit (SKU) number), item name, item cost, etc.).

FIG. 10 shows an example of a receipt structure 1000 in accordance withsome embodiments. The receipt structure 1000 is show as an overlay onthe receipt 902 to illustrate the associated receipt locations (e.g.,X-Y coordinate) for each receipt element. Example receipt elements mayinclude transaction receipt elements, category receipt elements, anditem receipt elements. The transaction receipt elements refer to receiptelements defining data about the transaction of the receipt, such asstore location receipt element 1002, store contact info receipt element1020, transaction date receipt element 1004, and transaction timereceipt element 1006. The category receipt elements, when used by areceipt, refers to receipt elements that define item categories such asentertainment-electronics receipt element 1008, grocery receipt element1010, and miscellaneous receipt element 1012.

The item receipt elements define individual items that were purchased,such as item receipt elements 1014 (e.g., US Weekly magazine) and 1016(e.g., broccoli). Receipt elements may also include sub-elements. Forexample, item receipt element 1016 may further include a SKU/identifierreceipt element (e.g., 21140319), an item type receipt element (e.g.,BROCCOLI), an item brand receipt element (e.g., BROCCOLI FLORETS), aunit size receipt element (e.g., 12 OZ), a unit count receipt element(e.g., 2@), a per unit price receipt element (e.g., $2.94), and a totalitem price receipt element (e.g., $5.88).

In some embodiments, different merchants may be associated withdifferent receipt structures, each including receipt elements andassociated locations where the corresponding fields are printed. Thereceipt structures may be stored by the receipt processing system 102and accessed for parsing extracted text, such as within database 106.Thus, the visual logo identification and merchant determination can beused to identify the appropriate receipt structure.

At 608, the one or more servers 104 (e.g., the receipt text extractor214) may be configured to perform text extraction of the receipt togenerate receipt text data. The receipt text data may refer to machinedata defining text that has been extracted from the image data of thereceipt using machine optical text extraction techniques, such as OCR.The receipt text extractor 214 may be configured to perform aline-by-line extraction of the text to generate the receipt text data,or a receipt element-by-recent element extraction of the text. FIG. 11shows an example of receipt text data 1100 extracted from the text 906of the image data 900 using a line-by-line extraction, in accordancewith some embodiments.

At 610, the one or more servers 104 (e.g., the receipt text extractor214) may be configured to associate the receipt text data with thereceipt elements of the receipt structure to create structured receiptdata. For example, the locations on the receipt 900 from where thereceipt text data 1100 is extracted may be compared with the predefinedlocations of the receipt elements as shown in FIG. 10 . Forcorresponding locations, the receipt text data may be associated withthe corresponding receipt element of the receipt structure. As such,structured receipt data including receipt text data that has beendefined with respect to a receipt structure is created.

In some embodiments, the receipt structure and structured receipt datamay be represented using a receipt structure language. The receiptstructure language may be implemented using the Extensible MarkupLanguage. For an example including the broccoli item, structured receiptdata may be defined using the receipt structure language as shown below:

<receipt>  <transaction>   <store>CHICAGO DIVISION</store>  <phone>312-460-8944</phone>   <date>07/16/2015</date>   <time>05:58PM</time>  </transaction>  <items>   <category>GROCERY</category>   <item>     <SKU>21140319</SKU>     <type>BROCCOLI</type>    <brand>BROCCOLI FLORETS</brand>     <unitsize>12 OZ</unitsiz>    <unitcount>2@</unitcount>     <unitprice>$2.94</unitprice>    <price>$5.88</price>    </item>  </items> </receipt>

Although only a single item of the receipt is shown, each of the itemslisted on the receipt may be associated with an <item></item> element ofthe receipt structure language. At 612, the one or more servers 104(e.g., the receipt text extractor 214) may be configured to determine aformat rule associated with a receipt element of the receipt structure.At 614, the one or more servers 104 (e.g., the receipt text extractor214) may be configured to update the structured receipt data associatedwith the receipt element based on the format rule if the structuredreceipt data violates the format rule.

A receipt element may be associated with format rules for the text dataof the receipt element. For example, a price receipt element may includea format rule specifying that the characters must be a number (e.g., thenumber 0 instead of the letter o), and/or begin with either a $ (e.g.,rather than the letter “S”) or a number (e.g., if the receipt does notuse $). As such, if an OCR error results in an alphabet character beingincluded within the price receipt element, the error may be identifiedand corrected based on the format rules.

Advantageously, the use of structured receipt data including receiptelements with format rules provides for improved OCR error correction by“simplifying” the problem of machine identification of similar, butdifferent, characters. For example, OCR techniques often have difficultydifferentiating the character $ and the letter s, or the letter b andthe number 6, etc. This may be particularly difficult in the context ofreceipt OCR because receipts are typically not designed to facilitateOCR reading (e.g., unlike checks), such as by using non-standard fontsrather than OCR fonts. The receipt text extractor 214 may be configuredto apply format rules to or each unit of structured receipt dataassociated with receipt elements, and updating as appropriate.

In some embodiments, receipt structures may be programmaticallydiscovered based on OCR. As such, the determination of the receiptstructure at 606 is not necessarily limited to manual operatordefinition. For example, applying OCR to image data of the merchant asdiscussed herein, the receipt text extractor 214 may be furtherconfigured to detect receipt elements and their associated (e.g., XYcoordinate) locations on the receipt. The receipt elements may bediscovered based on applying the format rules of the receipt elements tothe extracted receipt text data. For example, price receipt element maybe discovered based on detecting a “$” followed by a numerical value. Anitem receipt element may be identified based on a sequence of numbersrepresenting the SKU. And so forth. In another example, predefinedreceipt element values may be used, such as category names to discoverthe category receipt elements, known store locations to discover thelocation receipt element, item names/SKUs to discover item receiptelements, etc. Method 600 may then proceed to 616 and end.

Conclusion

Many modifications and other embodiments will come to mind to oneskilled in the art to which these embodiments pertain having the benefitof the teachings presented in the foregoing descriptions and theassociated drawings. For example, while the image cleanup, logoidentification, and text extraction techniques are discussed herein withrespect to an image of a receipt, they may be extended to other types ofdegraded documents. Degradation may refer to image quality, such as maybe caused by variances in camera quality, camera placement (e.g.,distance, gaze angle), lighting, image skew, document quality for OCR(e.g., folding, crumpling, non-optimal fonts, etc.). Therefore, it is tobe understood that embodiments and implementations are not to be limitedto the specific example embodiments disclosed and that modifications andother embodiments are intended to be included within the scope of theappended claims. Although specific terms are employed herein, they areused in a generic and descriptive sense only and not for purposes oflimitation.

That which is claimed:
 1. A system, comprising: one or more serversconfigured to: receive, from a consumer device, image data associatedwith a receipt; cause execution of a first image cleanup with respect tothe image data associated with the receipt; determine, based on thefirst image cleanup, a merchant logo associated with a merchant;determine, based on the merchant logo, a receipt structure associatedwith the merchant; cause execution of a second image cleanup withrespect to the image data associated with the receipt; extract, based onthe second image cleanup, one or more portions of receipt text data fromthe image data; determine a text match score based on a comparison ofthe one or more portions of receipt text data and one or more knownkeywords associated with the receipt structure related to the merchant;determine whether the text match score satisfies the text match scorethreshold; and in response to determining that the text match scoresatisfies the text match score threshold, generate structured receiptdata based on the one or more portions of receipt text data.
 2. Thesystem of claim 1, wherein the first image cleanup comprises a firstalgorithmic transformation of one or more pixels comprised in the imagedata based on a first set of predefined image parameters, and whereinthe second image cleanup comprises a second algorithmic transformationof one or more pixels comprised in the image data based on a second setof predefined image parameters.
 3. The system of claim 2, wherein thefirst set of predefined image parameters and the second set ofpredefined image parameters comprise at least one of: contrast,brightness, filter size, offset, smoothing strategy, enhancementstrategy, rotation, or skew.
 4. The system of claim 2, wherein the firstimage cleanup and the second image cleanup comprises performing one ormore of: an image de-skew, an image resize, or an image de-warp.
 5. Thesystem of claim 1, wherein the first image cleanup and the second imagecleanup are performed iteratively based on a predefined number ofiterations.
 6. The system of claim 5, wherein the one or more serversare further configured to: in response to determining that the merchantlogo associated with the merchant cannot be determined based onperforming the first image cleanup for the predefined number ofiterations: cause the consumer device to capture second image dataassociated with the receipt; and cause execution of the first imagecleanup with respect to the second image data.
 7. The system of claim 1,wherein the text match score threshold is associated with a particularmerchant, and wherein the text match score threshold is set for apredefined level of tolerance associated with the particular merchant.8. The system of claim 1, wherein the one or more portions of receipttext data can be categorized based on at least one of a field or asubject matter.
 9. The system of claim 1, wherein the structured receiptdata comprises one or more portions of fine receipt data associated withone or more items, and wherein the one or more portions of fine receiptdata comprise at least one of: an item name, an item stock keeping unit(SKU), an item price, or an item quantity.
 10. A method comprising:receiving, by one or more servers, image data associated with a receiptfrom a consumer device; causing, by the one or more servers, executionof a first image cleanup with respect to the image data associated withthe receipt; determining, by the one or more servers, based on the firstimage cleanup, a merchant logo associated with a merchant; determining,by the one or more servers, based on the merchant logo, a receiptstructure associated with the merchant; causing, by the one or moreservers, execution of a second image cleanup with respect to the imagedata associated with the receipt; extracting, by the one or moreservers, based on the second image cleanup, one or more portions ofreceipt text data from the image data; determining, by the one or moreservers, a text match score based on a comparison of the one or moreportions of receipt text data and one or more known keywords associatedwith the receipt structure related to the merchant; determining, by theone or more servers, whether the text match score satisfies a text matchscore threshold; and in response to determining that the text matchscore satisfies the text match score threshold, generating, by the oneor more servers, structured receipt data based on the one or moreportions of receipt text data.
 11. The method of claim 10, wherein thefirst image cleanup comprises a first algorithmic transformation of oneor more pixels comprised in the image data based on a first set ofpredefined image parameters, and wherein the second image cleanupcomprises a second algorithmic transformation of one or more pixelscomprised in the image data based on a second set of predefined imageparameters.
 12. The method of claim 11, wherein the first set ofpredefined image parameters and the second set of predefined imageparameters comprise at least one of: contrast, brightness, filter size,offset, smoothing strategy, enhancement strategy, rotation, or skew. 13.The method of claim 11, wherein the first image cleanup and the secondimage cleanup comprises performing one or more of: an image de-skew, animage resize, or an image de-warp.
 14. The method of claim 10, whereinthe first image cleanup and the second image cleanup are performediteratively based on a predefined number of iterations.
 15. The methodof claim 14, the method further comprising: in response to determiningthat the merchant logo associated with the merchant cannot be determinedbased on performing the first image cleanup for the predefined number ofiterations: causing, by the one or more servers, the consumer device tocapture second image data associated with the receipt; and causing, bythe one or more servers, execution of the first image cleanup withrespect to the second image data.
 16. The method of claim 10, whereinthe text match score threshold is associated with a particular merchant,and wherein the text match score threshold is set for a predefined levelof tolerance associated with the particular merchant.
 17. The method ofclaim 10, wherein the one or more portions of receipt text data can becategorized based on at least one of a field or a subject matter. 18.The method of claim 10, wherein the structured receipt data comprisesone or more portions of fine receipt data associated with one or moreitems, and wherein the one or more portions of fine receipt datacomprise at least one of: an item name, an item stock keeping unit(SKU), an item price, or an item quantity.
 19. A computer programproduct comprising at least one non-transitory computer-readable storagemedium having computer-executable program code instructions storedtherein, the computer-executable program code instructions comprisingprogram code instructions configured to: receive, from a consumerdevice, image data associated with a receipt; cause execution of a firstimage cleanup with respect to the image data associated with thereceipt; determine, based on the first image cleanup, a merchant logoassociated with a merchant; determine, based on the merchant logo, areceipt structure associated with the merchant; cause execution of asecond image cleanup with respect to the image data associated with thereceipt; extract, based on the second image cleanup, one or moreportions of receipt text data from the image data; determine a textmatch score based on a comparison of the one or more portions of receipttext data and one or more known keywords associated with the receiptstructure related to the merchant; determine whether the text matchscore satisfies the text match score threshold, and in response todetermining that the text match score satisfies the text match scorethreshold, generate structured receipt data based on the one or moreportions of receipt text data.
 20. The computer program product of claim19, wherein the first image cleanup comprises a first algorithmictransformation of one or more pixels comprised in the image data basedon a first set of predefined image parameters, and wherein the secondimage cleanup comprises a second algorithmic transformation of one ormore pixels comprised in the image data based on a second set ofpredefined image parameters.